Train-Test-Split, Data Leakage, and Other Considerations

Strategy Optimization Essentials

Effective strategy optimization in AI involves understanding key concepts and practices that ensure robust model performance:

Data Splitting:
- Divide the dataset into training and test sets.
- The model learns from the training set, while the test set evaluates performance.
- Avoid overfitting by preventing memorization of training patterns.
- Typical split allocation ranges from 10-30% for the test set.
Shuffling and Splitting:
- Randomly shuffle data to prevent bias, except for time series data where a temporal split is ideal.
Data Leakage:
- Ensure no target variable or its derived information appears in the training set, a danger known as target leakage.
- Impute missing values only post-split to avoid subtle leakage.
Class Imbalance:
- Address imbalances using techniques like oversampling.
- Use SMOTE (Synthetic Minority Oversampling Technique) on the training set only.
Domain-specific Insights:
- Pay attention to non-stationarity, regime changes, and survivorship bias for financial data.
- Avoid data snooping bias by not overfitting models to historical data patterns.

SOLUTION:

In a classification task, when 75% of the targets in the training set belong to one class, we should use oversampling techniques to address this.
The model should never see data from the test set during training.
With time series data, we do not shuffle the dataset prior to splitting it.